Repository navigation
feat(contexts): data model for grid pages - #3131
rubenvdlinde wants to merge 1 commit into
Conversation
9467b93 to
2f8eafc
Compare
4489199 to
5d26395
Compare
| $table->addColumn('icon', Types::STRING, ['notnull' => false, 'length' => 64]); | ||
| $table->addColumn('description', Types::TEXT, ['notnull' => false]); |
There was a problem hiding this comment.
Do we need an icon and a description on widgets now?
There was a problem hiding this comment.
Yes, I'd like to keep them. Pages and menu items have an optional icon and description too, so all three use the same icon picker and the same fields. The frontend shows them in the widget header. Both are optional and stay NULL when nobody sets them.
| $table->addColumn('grid_y', Types::INTEGER, ['notnull' => true, 'default' => 0]); | ||
| $table->addColumn('grid_width', Types::SMALLINT, ['notnull' => true, 'default' => 1]); | ||
| $table->addColumn('grid_height', Types::SMALLINT, ['notnull' => true, 'default' => 1]); | ||
| $table->addColumn('settings', Types::TEXT, ['notnull' => false]); |
There was a problem hiding this comment.
i thought we said we're using JSON?
There was a problem hiding this comment.
We did, sorry. The column is Types::JSON now. Nextcloud maps it per database: json on PostgreSQL, JSON on MySQL, LONGTEXT on MariaDB and text on SQLite. I ran the tests on PostgreSQL and SQLite; nothing compares the stored text, so MySQL normalising the JSON doesn't matter. The Widget entity maps the field as Types::JSON too and reads and writes it as an array. That part is in #3133, where the settings type moves into ResponseDefinitions anyway.
| $technicalNames = $this->importedPageNames($context['pages']); | ||
| foreach ($context['pages'] as $pageKey => $pageArr) { |
There was a problem hiding this comment.
The page type from the import file isn't checked. Hwo are we sure we're only accepting startpage and grid? Can someone create two start pages, for example?
There was a problem hiding this comment.
Good catch. An import now only takes startpage and grid pages, and at most one start page per application. Any other page is skipped with a warning in the log, like other invalid import data. There's a test with a second start page, an unknown type, a page without a type and one that isn't an object.
| * @method setContextId(int $value): void | ||
| * @method getPageType(): string | ||
| * @method setPageType(string $value): void | ||
| * @method getTechnicalName(): string |
There was a problem hiding this comment.
Yes. The column allows NULL for the moment between inserting a page and giving it page_<id>, so the getter and setter are ?string now, like the other nullable fields.
There was a problem hiding this comment.
In this file and other test file, it seems like we have more test code than feature code. Most of the extra comes from the same case being tested in two or three places, or from tests that check mocks or config instead of behaviour. We should cut out some tests or merge. Having too many tests which are not useful slows down CI and adds clutter
There was a problem hiding this comment.
Agreed, there was too much. I went through the tests of #3131, #3132 and #3133: one test per behaviour, cases that were tested in two or three places kept in one, near duplicates merged into data providers, and tests that only checked mocks, config or getters and setters removed. For this PR that's 36 test methods down to 22, and 817 test lines down to 691.
#3132
e0824c0 to
d20afd0
Compare
Pages get a title, a named icon, a description and a technical name that is unique within their application. Without a name a page is called page_<id>, the start page dashboard, and page_<number> is kept for those automatic names. Widgets get their own table: the page, the type, an optional title, icon and description, the position and size on the grid, and the settings as one JSON object. A widget source links a data widget to a table or view of the same application. One migration adds all of it and names the pages that already exist. The technical name format is checked by one validator, now also used for views and columns. The app version moves to 2.5.0-dev.0 so that instances on 2.4.0 run the migration. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
d20afd0 to
ada0004
Compare
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
blizzz
left a comment
There was a problem hiding this comment.
Blockers are the JSON type (should be TEXT, see dedicated comment), and the question about the technical_name, as well as the related import logic.
|
|
||
| protected ?int $contextId = null; | ||
| protected ?string $pageType = null; | ||
| protected ?string $technicalName = null; |
There was a problem hiding this comment.
Same concern for technical names as noted about menu entries in #3100 (comment)
| $table->addColumn('grid_y', Types::INTEGER, ['notnull' => true, 'default' => 0]); | ||
| $table->addColumn('grid_width', Types::SMALLINT, ['notnull' => true, 'default' => 1]); | ||
| $table->addColumn('grid_height', Types::SMALLINT, ['notnull' => true, 'default' => 1]); | ||
| $table->addColumn('settings', Types::JSON, ['notnull' => false]); |
There was a problem hiding this comment.
It can store Json, but it has to be TEXT. JSON is also deprecated as of NC 33: https://github.com/nextcloud/server/blob/b117761f029381403cdba2e647c1f43fb6bd73fc/lib/public/DB/Types.php#L177-L180
| $rows = $result->fetchAllAssociative(); | ||
| $result->closeCursor(); | ||
|
|
||
| $names = self::backfillNames($rows); |
There was a problem hiding this comment.
In theory, we can run into a memory issue here, hence better to deal with one result row at a time. Now, I am not expecting huge amounts of data there, should be fine hence.
| $names = []; | ||
| $taken = []; | ||
| foreach ($pages as $key => $pageArr) { | ||
| $name = is_array($pageArr) ? ($pageArr['technical_name'] ?? $pageArr['technicalName'] ?? null) : null; |
There was a problem hiding this comment.
hm where does the disparity between technical_name and technicalName come from? Same about the page type below.
| } | ||
| foreach ($pages as $key => $pageArr) { | ||
| $isStartPage = is_array($pageArr) && ($pageArr['page_type'] ?? $pageArr['pageType'] ?? '') === Page::TYPE_STARTPAGE; | ||
| if ($names[$key] === null && $isStartPage && !isset($taken[Page::START_PAGE_NAME])) { |
There was a problem hiding this comment.
Imported pages should be referenced by their exported id, not by technical name. The menu export will refer to pages by technical name. But importedPageNames() renames pages on import:
- it drops
page_<n>names and duplicates; dashboardcan move to another page
So looking up a page by name after import can return the wrong page or none at all. importContext() already builds $pageMap keyed by the exported page id, the same way $nodeRelMap works. Pages are always created new on import, so the exported id is enough to find them. Using it also handles the concern that technical names are mutable aliases and shouldn't serve as identifiers.
| $this->deleteByWidgetIds([$widgetId]); | ||
| if ($nodeRelId === null) { | ||
| return; | ||
| } | ||
| $qb = $this->db->getQueryBuilder(); | ||
| $qb->insert($this->table) | ||
| ->values([ | ||
| 'widget_id' => $qb->createNamedParameter($widgetId, IQueryBuilder::PARAM_INT), | ||
| 'node_rel_id' => $qb->createNamedParameter($nodeRelId, IQueryBuilder::PARAM_INT), | ||
| ]) | ||
| ->executeStatement(); |
AndyScherzinger
left a comment
There was a problem hiding this comment.
Review focused on performance, accessibility, security, clean code and architecture. The inline comments only cover points that aren't in the existing threads. This PR has no UI of its own; the one accessibility point is a data-model gap that shows up in the page bar of the stacked UI (see the comment on the page import).
|
|
||
| ]]></description> | ||
| <version>2.4.0</version> | ||
| <version>2.5.0-dev.0</version> |
There was a problem hiding this comment.
#2476 bumps to the same 2.5.0-dev.0 and ships its own migration (Version2500Date20261031000000). The two branches merge without a conflict on this line, so whichever lands second leaves the version unchanged, and Nextcloud only upgrades an app when the info.xml version differs from installed_version (AppManager::isUpgradeRequired()).
On an instance that already runs the first one (dev or nightly setups tracking main), Version2500Date20261022000000 then stays pending until the next bump: tables_contexts_page has no technical_name column, and ContextService::insertPage() now always writes it, so creating any application fails with an unknown-column error. A dev instance that ran #2476 is in exactly that state (installed_version 2.5.0-dev.0, page table without technical_name). In the other merge order, #2476's archive tables are the ones missing.
Whichever of the two lands second needs to bump past what main has at that point, and #3135 then one higher.
There was a problem hiding this comment.
One more thing on the ordering, checked on a dev instance running #2476: it has already applied Version2500Date20261031000000 and Version2500Date20261101000000, while this PR's migration is Version2500Date20261022000000, so it sorts before both. Whichever lands second, please rename it to the then-current target version and a current date, as AGENTS.md asks for long-lived PRs, so it sorts after everything already applied.
| foreach ($pages as $pageKey => $pageArr) { | ||
| $page = new Page(); | ||
| $page->setContextId($newContext->getId()); | ||
| $page->setPageType($pageArr['page_type'] ?? $pageArr['pageType'] ?? ''); |
There was a problem hiding this comment.
Grid pages are importable, but this loop never gives them a title. From #3132 on, the import keeps the title NULL whenever the exported one is missing, blank or too long, while the API (PageService::validateTitle() in #3133) requires 1 to 200 characters for a grid page.
In the page bar of the UI stack (ContextPageBar.vue), pageTitle() returns '' for such a page, so its role="tab" button contains only the aria-hidden icon and an empty label. Screen readers announce an unnamed tab, and sighted users see an unlabeled icon (WCAG 4.1.2 Name, Role, Value).
Giving an imported grid page without a usable title a fallback title (its technical name, for example), or skipping it like other invalid page entries, keeps the stored data within what the API allows. A fallback label in pageTitle() would cover the UI side.
| if ($schema->hasTable(self::PREFIX . 'widget_source')) { | ||
| return; | ||
| } | ||
| $table = $schema->createTable(self::PREFIX . 'widget_source'); |
There was a problem hiding this comment.
The source is 0..1 per widget (primary key on widget_id), so would a nullable, indexed node_rel_id column on tables_contexts_widget do the same job? Both reasons from #3100 hold for a column too: it's still a real column the server can check and clean up, and the own-application check (filterNodeRelIdsOfPageContext() in #3132) works the same way. The menu table in #3135 already stores its node_rel_id that way, as a nullable, indexed column on the row itself.
With the separate table, every read needs a second query or a join, and syncPage() in #3132 calls setSource() for every listed widget on every save: an unchanged page with N widgets still runs N DELETEs plus one INSERT per data widget, where an unchanged column would be skipped together with the widget's own UPDATE. A column would also make WidgetSource, WidgetSourceMapper and the delete-then-insert in setSource() unnecessary.
| * @param array<array-key, mixed> $pages exported pages | ||
| * @return array<array-key, array<array-key, mixed>> | ||
| */ | ||
| private function importablePages(array $pages): array { |
There was a problem hiding this comment.
This keeps at most one start page, but nothing guarantees one. An archive whose application has only grid pages (or no pages) imports as an application without a start page. Adding a table to it later goes through addNodeRelToPage(), which finds no start page and passes null to PageContentMapper::findByPageAndNodeRelation(int $pageId, ...), so the request ends in a TypeError (500) after the node relation row is already written, because update() isn't atomic.
Genuine exports always contain a start page, so this needs an edited or damaged archive, and it behaved the same before this PR. Since the import validates pages here now, it may be worth closing. A fallback would have to cover a missing pages key, can't assume dashboard is free (a grid page may keep that name), and should put the imported node relations on the new start page.
| $qb = $this->db->getQueryBuilder(); | ||
| $qb->select('*') | ||
| ->from($this->table) | ||
| ->where($qb->expr()->in('page_id', $qb->createNamedParameter($pageIds, IQueryBuilder::PARAM_INT_ARRAY))) |
There was a problem hiding this comment.
AGENTS.md: "IN clauses must be chunked to at most 1 000 items for Oracle compatibility. Use a named constant (DB_CHUNK_SIZE = 1_000)", as ContextMapper, ViewMapper, TableMapper, Row2Mapper and RowSleeveMapper do. None of the five new IN queries chunk: findByPageIds() and deleteByIds() here, and findNodeRelIdsByWidgetIds(), deleteByWidgetIds() and deleteByNodeRelIds() in WidgetSourceMapper.
In the stack, deleting or exporting an application passes every widget id of that application. With 100 widgets per page and no page limit, more than 1000 ids are reachable, and the server then logs "More than 1000 expressions in a list are not allowed on Oracle." at error level on every database. When chunking findNodeRelIdsByWidgetIds(), write each chunk into the same map: array_merge(...) would renumber the widget-id keys.
Related, in #3132: WidgetService::deleteByPageIds() calls findIdsByPageId() once per page with array_merge inside the loop, which AGENTS.md also rules out. findByPageIds() already covers it in one query.
| return (object)$settings; | ||
| } | ||
|
|
||
| public function jsonSerialize(): array { |
There was a problem hiding this comment.
Nothing in the stack calls this outside the tests. The API and export shape is built by hand in WidgetService::findByPageIds() (#3132) and differs from this one: it has source and no pageId, matching TablesWidget in #3133. Dropping JsonSerializable and asserting getSettingsObject() directly in WidgetTest and WidgetMapperTest would leave one widget shape.
| public function insertWithDefaultName(Page $page): Page { | ||
| return $this->atomic(function () use ($page): Page { | ||
| $page = $this->insert($page); | ||
| $page->setTechnicalName(Page::DEFAULT_NAME_PREFIX . $page->getId()); |
There was a problem hiding this comment.
QBMapper::insert() doesn't reset the updated fields, so the update() below re-sends every field set before the insert (context_id, page_type, ...) together with technical_name. Resetting first narrows it to SET technical_name = ?, the same way ContextService::create() resets after its insert.
| $page->setTechnicalName(Page::DEFAULT_NAME_PREFIX . $page->getId()); | |
| $page->resetUpdatedFields(); | |
| $page->setTechnicalName(Page::DEFAULT_NAME_PREFIX . $page->getId()); |
| /** | ||
| * A database error with a reason, as the database layer throws it. | ||
| */ | ||
| class DbErrorStub extends Exception { |
There was a problem hiding this comment.
Nothing in this PR uses this class; only bootstrap.php requires it, and its first user is PageServiceTest in #3133. Moving the file and the require_once there keeps this PR to the data model. A createMock(Exception::class) with getReason() stubbed would also work, since getReason() isn't final.
| @@ -386,11 +387,7 @@ public function importScheme(int $contextId, string $name, string $iconName, str | |||
| } | |||
|
|
|||
| protected function isValidIcon(string $iconName): bool { | |||
There was a problem hiding this comment.
Now that the rule lives in IconHelper, this method only forwards to it. Nothing overrides or mocks it, and the later PRs call IconHelper::isValid() directly, so the two callers (lines 114 and 155) could do the same and the method could go.
| } | ||
|
|
||
| private function assertValid(string $technicalName): void { | ||
| $method = new \ReflectionMethod(ViewService::class, 'assertTechnicalNameValid'); |
There was a problem hiding this comment.
This reaches the private assertTechnicalNameValid() through reflection on an instance built without its constructor, so renaming or inlining the helper breaks the test without any change in behaviour. ViewServiceAuditLogTest already builds a ViewService from mocks; calling create(..., technicalName: ...) there would keep both message assertions, including the length message now built from MAX_LENGTH, on the public path.
Part of #3099. This adds the data model from #3100, and nothing uses it yet. It's the first of seven pull requests that build the backend for grid pages and application menus. They're stacked, each on the one before:
Splitting it this way lets you review the schema on its own, before any behaviour depends on it.
Size: 1380 lines: 681 code and 691 tests, plus the version bump.
What changes
tables_contexts_page) gettitle,iconanddescription. A new page typegridsits next tostartpage.dashboard.tables_contexts_widget): the page, the type, an optional title, icon and description, the position and size on the 12-column grid, and the settings in oneJSONcolumn. Settings are read back ordered by name, because MySQL'sJSONcolumn type does not keep the order they were written in, so every database answers the same.tables_contexts_widget_source): the link from a data widget to a node of its own application. It's a real column, so the server can check it and clean it up.startpageandgridpages only, and at most one start page per application. Other pages are skipped with a warning in the log, like other invalid import data.One migration,
Version2500Date20261022000000, adds the columns and tables, including the unique index on the page's application and technical name, and then fills in a technical name for every existing page. Tables runs on PostgreSQL, MySQL and SQLite, which all treat NULL values in a unique index as distinct (not empty strings, which the code never writes), so the index can be created before the names are filled in. It also serves lookups by application, so there's no separate index oncontext_id. The column itself allows NULL: a new page gets itspage_<id>right after it's inserted, the same way views get theirs.Where it differs from #3100
We agreed these with @enjeck:
And two small ones on top:
iconinstead of anemoji. It's a named icon from the same set applications use, so applications, pages, menu items and widgets share one icon picker.How the technical names work
They work the same way as for views and columns:
page_<id>, likeview_<id>andcolumn_<id>. The start page of an application isdashboard.page_<number>is reserved for the automatic names, the way columns keep reserved names, so a name someone types can never collide with a later automatic one. The API refuses it from feat(api): pages and widgets on api/2 #3133 on; imports treat it as no name.^[a-z][a-z0-9_]*$, at most 200 characters, no trailing newline), now also used by views and columns, so the rule lives in one place.Why nothing breaks
The new tables start empty, and the new page columns are empty, except the technical name. Existing start pages get
dashboard, and new applications get the same when their start page is created. Nothing reads the rest yet, so existing applications, start pages and API responses stay as they are.The app version goes to
2.5.0-dev.0(ininfo.xml,package.jsonandpackage-lock.json, the way main went to2.4.0-dev.0after 2.3.2), so instances on the released 2.4.0 run the migration. The migrations of this stack are namedVersion2500Date…for the same reason.How it was tested
On Nextcloud 34 with PostgreSQL:
WidgetTest, plus new database testsWidgetMapperTest,WidgetSourceMapperTestandPageTestpage_<id>, filling in existing pages, refusing a name that's already used, a trailing newlineocc upgradeto this stack; both migrations ran, the start pages becamedashboardand another pagepage_<id>composer cs:check, psalm and rector, all clean🖼️ Screenshots
Backend only, no UI changes.
🏁 Checklist
/backport to stableX.X🤖 AI (if applicable)